Skip to content

feat(server): build, sign, and submit Bulletin preimages in the core#270

Open
pgherveou wants to merge 150 commits into
mainfrom
bulletin-preimage-in-core
Open

feat(server): build, sign, and submit Bulletin preimages in the core#270
pgherveou wants to merge 150 commits into
mainfrom
bulletin-preimage-in-core

Conversation

@pgherveou

@pgherveou pgherveou commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

What

Preimage submission to the Bulletin chain now happens entirely inside the Rust core (truapi-server).
The core builds, signs, and submits the TransactionStorage.store extrinsic itself, routing chain traffic through the host's existing chain.connect JSON-RPC pipe.

Previously the core handed the host a signing capability and the host built and submitted the transaction with PAPI.
That seam is gone. As a result:

  • the wallet-delegated allowance secret is only ever used inside the core: no signer handle and no sign request crosses the host / FFI boundary, and the key arrives from the wallet end-to-end encrypted
  • the host's only remaining preimage job is content lookup (smoldot bitswap, or an IPFS HTTP gateway in RPC-gateway mode);
  • all hosts (web, and later mobile over uniffi) get identical submission behaviour, because the logic lives in the shared core instead of each host.

Ownership: before vs after

The topology is unchanged (the core still reaches the network only through the host).
What changes is ownership.
Before, the secret bytes were core-owned but the capability to use them, and the decision of what gets signed, was host-owned:

BEFORE
         core-owned  (WASM worker)          host-owned  (JS main thread)
  +------------------------------+   ||   +------------------------------+
  | Rust core                    |   ||   | dotli (PAPI)                 |
  |                              |   ||   |                              |
  | owns:                        |   ||   | owns:                        |
  |  - allowance secret (64 B)   |   ||   |  - signer handle             |
  |  - sign() execution          |   ||   |     { publicKey, sign() }    |
  |                              |   ||   |  - tx builder (PAPI)         |
  |                              |   ||   |  - decides WHAT is signed    |
  |                              |   ||   |  - submission + watch        |
  +------------------------------+   ||   +------------------------------+

               core --[ signer handle ]--> host   (a live signing capability)
               core <--[ sign(payload) ]-- host   (one request per tx)
               core --[ signature      ]--> host --> Bulletin

After, everything signing-related sits on the core side; during submission the host just forwards opaque JSON-RPC text:

AFTER
         core-owned  (WASM worker)          host-owned  (JS main thread)
  +------------------------------+   ||   +------------------------------+
  | Rust core                    |   ||   | dotli                        |
  |                              |   ||   |                              |
  | owns:                        |   ||   | owns:                        |
  |  - allowance secret (64 B)   |   ||   |  - permission/confirm modals |
  |  - signer (crate-private,    |   ||   |  - chain.connect JSON-RPC    |
  |     never exported)          |   ||   |     pipe (forwards verbatim) |
  |  - tx builder (subxt)        |   ||   |  - lookupPreimage backend    |
  |  - decides WHAT is signed    |   ||   |     (bitswap / IPFS gateway) |
  |  - submission + watch        |   ||   |                              |
  +------------------------------+   ||   +------------------------------+

               core --[ JSON-RPC request  ]--> host --> Bulletin
               core <--[ JSON-RPC response ]-- host
               Bulletin pipe: opaque JSON-RPC text only,
               no signing capability, no sign requests

Submit flow

 PRODUCT  | CORE (holds keys, builds+signs)    : HOST (modals+pipe) : NET
 ---------+------------------------------------+--------------------+---
 submit   > receive                            :                    :
          | remotePermission                   > prompt user        :
          | confirmUserAction                  > prompt user        :
          | allowanceKey (cached, else SSO     :                    :
          |   via People statement store)      > fwd (E2E encrypted)> wlt
          | slotAccountKey (ciphertext)        < forward            < wlt
          | [[ 64-B key decrypted in core ]]   :                    :
          | chain.connect(bulletin)            > open pipe          :
          | chainHead_v1_follow                > forward            > N
          | Metadata_metadata_at_version *     > forward            > N
          | chainHead_v1_header (mortality)    > forward            > N
          | AccountNonceApi_account_nonce *    > forward            > N
          | [[ build + SIGN store(value) ]]    :                    :
          | TaggedTransactionQueue_            :                    :
          |   validate_transaction * (dry-run) > forward            > N
          | transaction_v1_broadcast           > forward            > N
          | watch: nonce probe * per block,    :                    :
          |   chainHead_v1_body on advance     > forward            > N
          | chainHead_v1_storage(Events)       > forward            > N
       key< return blake2_256(value)           :                    :

 * runtime calls, carried on the wire as chainHead_v1_call

Changed surface

  • host_logic/extrinsic.rs: offline subxt assembler (config-pinned SubstrateConfig, sr25519 signer, metadata / validity / events / header decoders).
  • host_logic/bulletin.rs: store{data} build + sign; the call is resolved by name from the fetched metadata, with a hard check that the data argument really is a byte sequence.
  • runtime/bulletin_rpc.rs: the submit flow above + typed errors.
  • runtime.rs: Preimage::submit ordering with one allowance refresh + retry on rejection; lookup_subscribe cache + integrity check.
  • truapi-platform: PreimageHost keeps only lookupPreimage; BulletinAllowanceKey zeroized on drop; host configs require the Bulletin genesis hash (runtimeConfig.bulletin.genesisHash on the TS side).
  • Host TS (@parity/truapi-host) + dotli submodule: signer bridge removed; dotli keeps content lookup only (smoldot bitswap_v1_get, or IPFS gateway in RPC-gateway mode) and serves Bulletin chain.connect on both its light-client and RPC-gateway backends.

pgherveou added 30 commits June 30, 2026 17:30
Adds the canonical testing module (api/testing.rs) and its v01/v02/versioned
wiring used by the Rust host runtime and generated clients.
New crate defining the host syscall traits (storage, navigation, consent,
permissions, ...) that host runtimes implement. Types are re-exported from
truapi::versioned/v01 rather than redefined.
WASM host runtime that hosts implement: dispatcher, SCALE frames, subscription
streams, chain runtime, host logic (sessions, SSO pairing, permissions,
statement store, dotns) and the wasm bindings. Includes the committed generated
dispatcher/wire-table under src/generated/.
…backs

Extends the rustdoc-JSON code generator to emit the Rust dispatcher and wire
table consumed by truapi-server, plus the TS host-callbacks adapter. Golden
tests pin the emitted shapes.
New WASM-backed host runtime package embedding the Rust core, with web iframe
and Web Worker entry points. Updates the @parity/truapi client (SCALE, sandbox,
transport) and drops the obsolete explorer 0.3.2 codegen snapshot.
Updates CLAUDE.md/README, CI workflows, Makefile, deny.toml, changesets, and
linguist attributes for generated code, and bumps the dotli submodule to the
host integration that consumes the WASM runtime.
Adds the canonical testing module (api/testing.rs) and its v01/v02/versioned
wiring used by the Rust host runtime and generated clients.
New crate defining the host syscall traits (storage, navigation, consent,
permissions, ...) that host runtimes implement. Types are re-exported from
truapi::versioned/v01 rather than redefined.
…backs

Extends the rustdoc-JSON code generator to emit the Rust dispatcher and wire
table consumed by truapi-server, plus the TS host-callbacks adapter. Golden
tests pin the emitted shapes.
pgherveou added 7 commits July 9, 2026 18:36
Move the entire Bulletin TransactionStorage.store submission into
truapi-server. The core builds the extrinsic offline (subxt 0.50.2), signs
it with the wallet-delegated allowance key, dry-runs it, broadcasts, and
watches for inclusion over the existing chainHead runtime — replacing the
host signer-callback seam so the allowance secret never crosses the
host/FFI boundary.

Core:
- host_logic/extrinsic.rs: offline SubstrateConfig assembler with a
  config-pinned genesis hash, an sr25519 Signer, and metadata / transaction
  validity / events / header decoders.
- host_logic/bulletin.rs: store{data} construction signed with the allowance
  key, with audited pallet/call-index pinning plus a canonical-bytes guard so
  provider metadata cannot redirect the signature, and a memcpy call-data
  encoder that avoids scale-encode's per-byte cost.
- runtime/bulletin_rpc.rs: serialized submit flow (ephemeral with_runtime
  follow, metadata, nonce, validate_transaction dry-run, broadcast, single
  event-loop inclusion watch gated on nonce advance, System.Events dispatch
  check), typed error taxonomy, and broadcast stop on every exit.
- runtime.rs: Preimage::submit gates on bulletin availability before any
  prompt and refreshes the allowance (Increase policy) with one retry on an
  allowance rejection; lookup_subscribe verifies blake2_256(value)==key and
  serves an in-core content-addressed cache.
- BulletinAllowanceKey is zeroized on drop; PreimageHost keeps only
  lookup_preimage; both host configs gain an optional Bulletin genesis hash.

Codegen/TS: regenerate goldens (product wire unchanged) and drop the signer
bridge from the handwritten host worker. CI compiles the crate for
wasm32-unknown-unknown; .gitignore ignores the renamed wasm bundle path.

Bumps the hosts/dotli gitlink to the matching submodule commit.
Adversarial review of the watch loop found two defects:

- A crafted or buggy chain provider could send a self-referential or cyclic
  `NewBlock` parent link. The nonce-advance ancestor walk had no visited-set
  guard and no `.await`, so such a link spun forever, freezing the worker and
  permanently holding the submit lock across all products. The walk is
  extracted into `ancestors_to_check`, which guards against self-parent and
  cyclic links with a visited set, and is unit-tested.
- Blocks that failed the nonce gate were marked checked but never unpinned,
  leaking chainHead pins over the watch's lifetime and risking a false
  BroadcastUnverified once the server's pin limit was hit. They are now
  unpinned like body-negative blocks.

Also log a warning when a host lookup value is downgraded to a miss for
failing the blake2_256(value)==key integrity check.
…bler

The signing-host role now builds and signs transactions locally instead of
returning Unavailable, reusing host_logic/extrinsic.rs.

Because ProductAccountTxPayload carries each extension's `extra` and
`additional_signed` already SCALE-encoded in canonical order, assembly is a
pure offline concatenation — no metadata, no RPC:

- extrinsic.rs gains build_signed_extrinsic_v4 + v4_signer_payload and an
  Sr25519Signer::from_keypair constructor. Body = Compact(len) ++ 0x84 ++
  MultiAddress::Id(signer) ++ MultiSignature::Sr25519(sig) ++ Σextra ++
  call_data; signer payload = call_data ++ Σextra ++ Σadditional_signed,
  blake2_256 only when >256 bytes. Layout is byte-identical to subxt /
  frame-decode.
- signing_host::create_transaction handles Product and LegacyAccount (with a
  fail-closed slot-zero key-match check); the product-facing entrypoint's
  caller-scoping, chain-submit permission, and user-confirmation gates already
  precede it.
- Extrinsic V5 (tx_ext_version != 0) returns the new AuthorityError::NotSupported
  -> HostCreateTransactionError::NotSupported: V5 general carries the signature
  inside a VerifySignature extension, which cannot come from pre-encoded parts.

Tested: v4 layout + signature verification, the >256 hashing boundary, extension
order preservation, and Product/LegacyAccount success plus v5/mismatch/no-session
rejections.
@pgherveou pgherveou force-pushed the bulletin-preimage-in-core branch from 0836131 to 60b43ea Compare July 10, 2026 13:22
@TarikGul TarikGul self-requested a review July 10, 2026 15:41
@pgherveou pgherveou changed the base branch from worktree-issue-96-rust-core-port to main July 13, 2026 09:11
@pgherveou pgherveou requested review from a team July 13, 2026 09:11
…core

# Conflicts:
#	.github/workflows/ci.yml
#	.github/workflows/release.yml
#	.gitignore
#	CLAUDE.md
#	Makefile
#	hosts/dotli
#	js/packages/truapi-host/scripts/build-wasm.mjs
#	js/packages/truapi-host/src/adapter-support.ts
#	js/packages/truapi-host/src/host-callbacks-adapter.test.ts
#	js/packages/truapi-host/src/runtime.ts
#	js/packages/truapi-host/src/web/create-iframe-host.test.ts
#	js/packages/truapi-host/src/web/create-iframe-host.ts
#	js/packages/truapi-host/src/web/create-worker-host-runtime.ts
#	js/packages/truapi-host/src/web/worker-provider.test.ts
#	js/packages/truapi-host/src/worker-protocol.ts
#	js/packages/truapi-host/src/worker-runtime.ts
#	js/packages/truapi/src/sandbox.ts
#	playground/src/lib/auto-test.ts
#	playground/tests/e2e/dotli-diagnosis.ts
#	playground/tests/e2e/dotli/helpers/signer-bot.ts
#	rust/crates/truapi-codegen/src/rust/wasm_bridge.rs
#	rust/crates/truapi-codegen/src/ts/host_callbacks.rs
#	rust/crates/truapi-codegen/tests/golden/host-callbacks-adapter.ts
#	rust/crates/truapi-codegen/tests/golden/host-callbacks.ts
#	rust/crates/truapi-codegen/tests/golden/wasm_bridge.rs
#	rust/crates/truapi-codegen/tests/golden/worker-callbacks.ts
#	rust/crates/truapi-codegen/tests/golden_rust_emit.rs
#	rust/crates/truapi-server/src/runtime.rs
#	rust/crates/truapi-server/src/wasm.rs
#	rust/crates/truapi-server/src/wasm/generated_bridge.rs
Keep physical provider ownership in hosts while moving shared chain behavior and Subxt-backed handling into Rust. Harden Bulletin retries and host lifecycle coverage.
let elapsed = start.elapsed();
assert!(signed.encoded().len() > data.len());
assert!(
elapsed < std::time::Duration::from_secs(5),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is why the CI is failing. Maybe consider bumping as it currently taking 6.7 sec

Suggested change
elapsed < std::time::Duration::from_secs(5),
elapsed < std::time::Duration::from_secs(10),

@valentinfernandez1 valentinfernandez1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good to me just some small nits.

/// build + dry-run + broadcast + best-block inclusion. Passed explicitly to the
/// chain layer, which uses the call context only for cancellation.
const PREIMAGE_SUBMIT_BUDGET: Duration = Duration::from_secs(180);
/// Preimage submission is exercised by host diagnosis with a 240s live-chain

@valentinfernandez1 valentinfernandez1 Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment seems more related to the truapi-playground diagnosis, not sure it makes much sense here.

people: {
genesisHash: string | Uint8Array;
};
/** Bulletin-chain genesis hash used for in-core preimage submission. */

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a blocker but let's make sure to mark this as a breaking change. Also can we add docs to the rest of the fields here? 🙏🏻

///
/// // Submit a preimage first so the lookup resolves to a value.
/// const submitted = await truapi.preimage.submit("0xdeadbeef");
/// const value = crypto.getRandomValues(new Uint8Array(4)).toHex() as `0x${string}`;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the previous example was better

///
/// ```ts
/// const result = await truapi.preimage.submit("0xdeadbeef");
/// const value = crypto.getRandomValues(new Uint8Array(4)).toHex() as `0x${string}`;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here

[target.'cfg(target_arch = "wasm32")'.dependencies]
futures-timer = { version = "3", features = ["wasm-bindgen"] }
js-sys = "0.3"
subxt = { version = "0.50.2", default-features = false, features = ["web"] }

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should measure the impact of this on the bundle size as I am guessing it will increase it significantly, not an issue on desktop and mobile but it might on the browser

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants